Showing posts with label Python. Show all posts
Showing posts with label Python. Show all posts

Friday, February 14, 2020

Python with Selenium Webdriver

## Python with Selenium Helper file :-
-----------------------------------------

##Check version/ Is already installed? via CMD
-----------------------------------------------
python --version
pip --version

CTRL + '/' => Comment or Uncomment any line 
CTRL + ALT + SHIFT => Formate Code automatically


##Install Selenium in Python via CMD
---------------------------------------
pip install -U selenium


##Upgrade PIP version via CMD
------------------------------------
python -m pip install --upgrade pip


##Learn Selenium Python : Step by Step
------------------------------------------------------
Step 1 : download python - https://www.python.org/downloads/
Step 2 : Install and check python and pip is installed successfully
              python --version
              pip --version
Step 3 : install selenium libraries
              pip install -U selenium
Step 4 : Download PyCharm - community edition
      https://www.jetbrains.com/pycharm/dow...
Step 5 : Create new project in PyCharm
Step 6 : Adding selenium scripts to the project
   note: those who face problem on click btnK replace it with "q".
Step 7 : Run from IDE
         Run from Command Line


For more deatils about sample project kindly refer a below URL:

https://github.com/autom99/PythonSampleProject/tree/master/PythonWithSelenium


Saturday, February 8, 2020

Python Web Scraping Using BeautifulSoup

Web scraping is the process of constructing an agent which can extract, parse, download and organize useful information from the web automatically. It is also called as web data mining or web harvesting.
In a simple word, Web scraping is used to collect large information from websites. 
Applications of web scraping:
  • Price Comparison: Services such as ParseHub use web scraping to collect data from online shopping websites including using it to compare the prices of products.
  • Email address gathering: Many companies that use email as a medium for marketing, use web scraping to collect email ID and then send bulk emails.
  • Social Media Scraping: Web scraping is used to collect data from Social Media websites such as Twitter to find out what’s trending.
  • Research and Development: Web scraping is used to collect a large set of data (Statistics, General Information, Temperature, etc.) from websites, which are analyzed and used to carry out Surveys or for R&D.
  • Job listings: Details regarding job openings, interviews are collected from different websites and then listed in one place so that it is easily accessible to the user.



Implementing Web Scraping in Python with BeautifulSoup


There are mainly two ways to extract data from a website:
  • Use the API of the website (if it exists). For example, Facebook has the Facebook Graph API which allows retrieval of data posted on Facebook.
  • Access the HTML of the webpage and extract useful information/data from it. This technique is called web scraping or web harvesting or web data extraction.
This article discusses the steps involved in web scraping using the implementation of Web Scraping in Python with Beautiful Soup
Steps involved in web scraping:


  1. Send an HTTP request to the URL of the webpage you want to access. The server responds to the request by returning the HTML content of the webpage. For this task, we will use a third-party HTTP library for python-requests.
  2. Once we have accessed the HTML content, we are left with the task of parsing the data. Since most of the HTML data is nested, we cannot extract data simply through string processing. One needs a parser which can create a nested/tree structure of the HTML data.
    There are many HTML parser libraries available but the most advanced one is html5lib.
  3. Now, all we need to do is navigating and searching the parse tree that we created, i.e. tree traversal. For this task, we will be using another third-party python library, Beautiful Soup. It is a Python library for pulling data out of HTML and XML files.
In below example, I am using PycharmIDE:

source code: TestWebScrap.py

-----------------------------------------------------------------
import requests
from bs4 import BeautifulSoup

page = requests.get("https://autom99.blogspot.com/")
soup = BeautifulSoup(page.content,'html.parser')
print(soup.prettify())

posts = soup.find_all(class_ ='posts')
print(posts)

postHierarchy = soup.find_all(class_ = 'hierarchy')
print(postHierarchy)

print(soup.find_all(id='ArchiveList'))
---------------------------------------------------------------------

Also, we can export the results into EXCEL as followed by the below code:

import pandas as pd
................-----------------

all = pd.DataFrame({
    'Posts': posts,
    'PostHierarchy': postHierarchy,
})
print(all)
------------------------


For more details refer to a URL my Github:







Thursday, January 16, 2020

Python Error solution 100% working

-------------------------------
Error description:
------------------------------
No module found error
any package installation failed  after installing by cmd

Go to PyCharm IDE -> theb
File -> Settings -> Projects -> Select '<Python installation path>' -> then 'Apply'.

then Run again test.

----------------------------------------------

Error messages in Python can often be confusing. Here is a list of common error messages you may find, along with a plain English explanation. These error messages are the result of runtime errors. They will immediately crash your Here are the error messages explained (your error messages may be slightly different but mean the same thing):

SyntaxError: invalid syntax
ImportError: No module named raandom
SyntaxError: EOL while scanning string literal
AttributeError: 'str' object has no attribute 'lowerr'
IndentationError: expected an indented block
IndentationError: unexpected indent
IndentationError: unindent does not match any outer indentation level
TypeError: bad operand type for abs(): 'str'
TypeError: abs() takes exactly one argument (2 given)
IndexError: list index out of range
KeyError: 'spam'
SyntaxError: invalid syntax
This is the most generic error message the Python interpreter will give you. It means that Python was expecting something that isn't there, or there is something there that it didn't expect. Maybe you forgot to include or inserted an extra character. Here are some examples:

if guess = 5:
In the above case, the programmer used = (the assignment operator) instead of == (the equals comparator operator). Python never expects assignment statements where there should be a condition.

def foo(:
In the above case, the programmer forgot to match the ending ) closing parenthesis.

def foo()
In the above case, the programmer forgot to put the colon at the end of the def statement. This can also happen with for, while, if, elif, and else statements.

ImportError: No module named raandom
This error shows up when you try to import a module that does not exist. Most likely, you have a typo in the module name. For example, you may have typed raandom instead of random.

SyntaxError: EOL while scanning string literal
print('Hello world!)
print("Hello world!')
This error happens when you do not have two quote marks for a string, or you use different quote marks for the same string. Look at these two examples:

AttributeError: 'str' object has no attribute 'lowerr'
'Hello'.lowerr()
'Hello'.append('x')
This error appears when you call a method or access an attribute that does not exist. This is most likely because 1) you have a typo in the method or attribute name, or 2) you are calling the method or attribute on a value that is the wrong data type. For example, strings have a method named lower(), but not lowerr() (that is a typo). And the append() method is a list method, so calling it on a string value will cause this error.

IndentationError: expected an indented block
def foo():
print('Hello world!')
This error happens if you fail to indent your code for a block. In the above example the print() call is at the same level of indentation as the def statement, when it should have a larger indentation.

IndentationError: unexpected indent
def foo():
    print('Hello world!')
     print('Goodbye')
An unexpected indent error happens when you add an indentation for no reason. You should only add indentation after a def, if, else, elif, while, or for statment (or any statement that ends with a colon.)

IndentationError: unindent does not match any outer indentation level
def foo():
    print('Hello world!')
   print('Goodbye')
This indentation error appears when you are decreasing the indentation, but not decreasing it to the same level as the previous indentation. The print('Goodbye') call should either be at the same indentation as the other print() call (and be inside the if block) or at the same indentation as the if statement (and be outside of the if block).

TypeError: bad operand type for abs(): 'str'
abs('Hello')
This error occurs when the value of an argument you pass to a function or method is of the wrong data type. In the above example, the abs() function takes an integer or floating point number. Passing a string for the argument results in an error.

TypeError: abs() takes exactly one argument (2 given)
abs(42, 50)
This error appears when you pass the wrong number of arguments to a function or method, either too many or too few. The abs() function takes exactly one (and only one) argument. In our example we pass two arguments, which results in this error.

IndexError: list index out of range
myList = ['spam', 'fizz', 'eggs']
print(myList[3])
The IndexError happens when the index you use is larger than or equal to the number of actual items in the list. In our above example, the myList list only has 3 items in it, so the only valid indexes to use are 0, 1, and 2. The index 3 (or any other index larger than 2) is larger than any of these indexes, so the code results in an IndexError.

KeyError: 'spam'
myDict = {'fizz':42, 'eggs':100}
myDict['spam']
The KeyError happens when you try to access a key in a dictionary object that does not exist. Either the key was never added to the dictionary, was deleted previously with the del operator, or the key you are using has a typo in it.


------------------------------------------------------------
Some Useful URL for error solving in Python
--------------------------------------------------------------

http://inventwithpython.com/appendixd.html

https://nostarch.com/automatestuffresources

https://nostarch.com/automatestuff2

https://pyautogui.readthedocs.io/en/latest/install.html



Python with Selenium Helper file

## Python with Selenium Helper file :-
-----------------------------------------

##Check version/ Is already installed? via CMD
-----------------------------------------------
python --version
pip --version

CTRL + '/' => Comment or Uncomment any line
CTRL + ALT + SHIFT => Formate Code automatically


##Install Selenium in Python via CMD
---------------------------------------
pip install -U selenium


##Upgrade PIP version via CMD
------------------------------------
python -m pip install --upgrade pip


##Learn Selenium Python in ONE VIDEO | Step by Step
------------------------------------------------------
Step 1 : download python - https://www.python.org/downloads/
Step 2 : Install and check python and pip is installed successfully
              python --version
              pip --version
Step 3 : install selenium libraries
              pip install -U selenium
Step 4 : Download PyCharm - community edition
      https://www.jetbrains.com/pycharm/dow...
Step 5 : Create new project in PyCharm
Step 6 : Adding selenium scripts to the project
note: those who face problem on click btnK replace it with "q".
Step 7 : Run from IDE
         Run from Command Line